Psychological Research Online: Opportunities and Challenges

As the Internet has changed communication, commerce, and the distribution of information, so too it is changing psychological research. Psychologists can observe new or rare phenomena online and can do research on traditional psychological topics more efficiently, enabling them to expand the scale and scope of their research. Yet these opportunities entail risk both to research quality and to human subjects. Internet research is inherently no more risky than traditional observational, survey or experimental methods. Yet the rapidly changing nature of technology, norms, and online behavior means that the risks and safeguards against them will differ from those characterizing traditional research and will themselves change over time. This paper describes some benefits and challenges of conducting psychological research via the Internet and offers recommendations to both researchers and Institutional Review Boards for dealing with the challenges. Send comments and editorial correspondence to: Robert Kraut HCI Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh PA 15213 [email protected] 412 268-7694 Psychological Research Online: Opportunities and Challenges Robert Kraut, Judith Olson, Mahzarin Banaji, Amy Bruckman, Jeffrey Cohen, Mick Couper, The Internet as a research vehicle presents both opportunities and challenges for psychological research. In 1985, only 8.2% of US households had a personal computer, and the Internet as we now know it, with its rich array of communication, information, entertainment, and commercial services, did not exist. Since then, this exotic technology has become domesticated and is now used by the majority of Americans for personal and economic reasons (Cummings & Kraut, 2002). By September of 2001, 66% of the US population used a computer at home, work, or school, and the vast majority of these, 56% of the US population, also used the Internet (U. S. Department of Commerce, 2002). The Internet and the widespread diffusion of personal computing have the potential for unparalleled impact on the conduct of psychological research. For example, the Internet has changed the way scientists collaborate, by increasing the ease with which they can work with geographically distant partners (Walsh & Maloney, 2002) or share information (e.g., http://www.socialpsychology.org/). In this article we will focus on the way the Internet is changing the process of empirical research. The Internet presents empirical researchers with opportunities. It lowers many of the costs associated with collecting data on human behavior, can host online experiments and surveys, allows observers to watch online behavior, and offers the mining of archival data sources. For example, online experiments can collect data from thousands of participants with minimal intervention on the part of experimenters (B. A. Nosek, M. Banaji, & A. G. Greenwald, 2002a). Internet chat rooms and bulletin boards provide a rich sample of human behavior that can be mined for studies of communication (Nardi & Whittaker, 2001), prejudice (Glaser, Dixit, & Green, 2002), organizational behavior (Orlikowski, 2000), or diffusion of innovation (Kraut, Rice, Cool, & Fish, 1998), among other topics. The Internet is also a crucible for observing new social phenomena, such as the behavior of very large social groups (Sproull, 1995), distributed collaboration (Hinds, 2002), and identity-switching (Turkle, 1997), which are interesting in their own right and have the potential to challenge traditional theories of human behavior. At the same time, the Internet raises substantial challenges in terms of quality of data and the treatment of research participants. For example, researchers often lose control over the context in which data are procured when subjects participate in experiments online. Insuring informed consent, explaining instructions, and conducting effective debriefings may be more difficult than in the traditional laboratory experiment. Observations in chat rooms and bulletin boards raise difficult questions about risks to participants, including privacy and lack of informed consent. This article will discuss both the advantages of this new mode for psychological research as well as the challenges that it poses to data quality and the protection of research participants. APA-Internet Version 3.3 9/30/2003 Page 3 After discussing the opportunities and challenges of conducting online research, we close with recommendations in light of these challenges, directed toward both the researcher and the Institutional Review Boards that oversee the protection of human research subjects. We focus our attention primarily on online experiments, surveys, and observation of naturally occurring online behavior, because these are the major types of research conducted currently by psychologists who use the Internet. Furthermore, these methods have obvious parallels in the off-line (non-Internet) world that can be used as yardsticks by which to compare the online methods. Opportunities of Internet research The Internet can have positive impact on the conduct of psychological research, both by changing the costs of data collection and by making visible interesting psychological phenomena that do not exist in traditional settings or are difficult to study there. Making empirical research easier Compared to other modes of collecting data, the Internet can make observational research, self-report surveys, and random-assignment experiments easier to conduct. This ease derives largely from two properties of Internet research: economy and access. Subject recruitment. Use of the Internet decreases the cost of recruiting large, diverse, or specialized samples of research participants for either surveys or online experiments. Many researchers attract volunteers by posting announcements at relevant web sites and distribution lists. This technique can provide a large a diverse sample at low cost. For example, in four years, Nosek, Banaji, and Greenwald (2002b) collected a data set of over 1.5 million completed responses in tests of implicit attitudes. (See Sidebar 2). A survey on online behavior collected data from 40,000 respondents from many countries (Wellman, Quan Haase, Witte, & Hampton, 2001), simply by putting a link to the survey on a National Geographic website. On a smaller scale, the research reported in Sidebar 4 (Williams, Cheung, & Choi, 2000) conducted a pair of online experiments about ostracism, with over 1,500 participants from over 60 countries. And those conducting usability tests of websites can merely post “try this new page and give us your reactions” on a busy website and get thousands of responses within hours. One can post a research opportunity at service sites that specialize in advertising the availability of such opportunities, such as the one hosted by the Social Psychology Network (http://www.socialpsychology.org/expts.htm) or the American Psychological Society (http://psych.hanover.edu/APS/exponnet.html). Commercial services, such as Survey Sampling, Inc. (http://www.surveysampling.com) are available to aid in selecting a sample. Alternately, one can invite participation by sending personalized electronic mail messages to active participants in either specialized or more general online communities (See Couper, Traugott, & Lamias, 2001 for a review of sampling approaches for Internet surveys.) In one sense, the Internet has democratized data collection. Researchers do not need access to introductory psychology classes to recruit research subjects and often do not need grant money to pay them.. The Internet has opened research to those with fewer APA-Internet Version 3.3 9/30/2003 Page 4 resources. One consequence is that faculty at small schools, independent scholars, graduate students, and undergraduates can all potentially contribute to psychological research. For example, an undergraduate psychology major, Nicholas Yee, published findings about the psychology of playing online multi-player games, based on 19 surveys he directed to players of the Internet game EverQuest between September 2000 and April 2001, collecting over 18,500 responses from approximately 3,300 players. However, a corollary of this open access is that those with minimal training and supervision can conduct and publish research, some of which might be of low quality. Yee’s research results, for example, are available on his own website (www.nickyee.com) but have not been published in any peer-reviewed venue. Regardless of the quality of this research, his intense polling of a single population has polluted this data source for researchers who may be more qualified. In this sense, the tragedy of the commons has now threatens psychological research (Hardin, 1968). In an another case, an undergraduate, Martin Rimm, published a study in the Georgetown Law Review (Rimm, 1995) reporting on the prevalence of pornography, using research methods that have been heavily disputed (Thomas, 1996). Observing social behavior. The Internet provides scientists interested in social behavior with many archives of communication, from online groups in discussing topics as diverse as medical support, hobbies, popular culture, and technical information (e.g., see the newsgroups archives at http://groups.google.com/groups or the collections of email-based distribution lists at http://tile.net/lists/). Researchers have used these online groups to study such social processes as personal influence (Cummings, Sproull, & Kiesler, 2002), negotiation (Biesenbach-Lucas & Weasenforth, 2002), and identity formation (McKenna & Bargh, 1998). Many online forums make visible psychological phenomena that would be much more difficult to study in traditional settings. Some phenomena, like the evolution of groups or long-term learning, are ordinarily difficult to study in controlled settings because of the difficulty of bringing subjects back to the laboratory many times. Research in social psychology on groups larger than three or four are again difficult to study in the laboratory. Studying large groups over time merely compounds these problems. The Internet has provided a new venue for such long-term research on large groups. For example, Baym (1998) was able to explore the way groups develop a sense of community over an extended time period, by examining the use of an electronic mail distribution list about soap operas. Similarly, Butler (2001) was able to study the impact of participation on the attraction and retention of group members, by creating an archive of all messages sent to 206 online groups over a three-month period. Finally, Bos et al. (2002) examined the development of social capital by having groups of up to 24 play a game on the Web, in which individuals exchanged favors at anytime they wished for a month. In contrast to conducting observational research in face-to-face settings, for example in a classroom or playground, where the researcher’s presence may contaminate the phenomenon under study, researchers can be less obtrusive when conducting observation online. Conducting research online, Bruckman (1999) was able to study the influence of groups on long-term learning, by tracking 475 children learning a programming language APA-Internet Version 3.3 9/30/2003 Page 5 over a five-year period. Furthermore, because the participants in online groups type their own comments and dialogue, the researcher no longer needs to transcribe the data. The researcher can use simple programs to perform content analyses, examining, for example, differences in different age groups or the ways boys and girls use the tools they are given (Bruckman, 1999). Access to other archival data. The records of individual behavior on the Internet can provide a source of detailed, unobtrusive data for other phenomena besides social behavior (Webb, Campbell, & Swartz, 1999). The detailed transaction logs that people leave when using the Internet for a wide variety of activities provide a wealth of potential data for study. These include browsing behavior, application use, purchasing behavior, file uploads and downloads, subscription to communication forums, email sending, and a host of other digital transactions. For example, both academic and market researchers have used the Internet as a source of data about individual preference and choice (Montgomery, 2001). Others have used the history of uploads and downloads of music files to document the extent of social loafing and the rarity of altruistic behavior online (Adar & Huberman, 2000). These records include information about sequences of behavior, not only their quantity. Because most online transactions have detailed time stamps, one can analyze sequences of behavior, observing how events early in a sequence influence those occurring later. For example, Hoffman, Novak, and Duhachek (2002) used the time sequence of online behavior to model the concept of psychological flow (Csikszentmihalyi & Csikszentmihalyi, 1988), and Kraut and his colleagues (Kraut, 1999) used records of Internet users’ email traffic to document changes in the geographic dispersion of social networks over a two-year period. Automation and experimental control. One of the benefits of online research is that it allows a degree of automation and experimental control that can be otherwise difficult to achieve without the use of computers. A primary advantage of the Internet for both survey and experimental research is the low marginal cost of each additional research participant. Unlike traditional laboratory experiments or telephone surveys, where each new participant must be encountered, instructed and supervised by a person, most online experiments and surveys are automated with a low marginal cost: a human experimenter does not need to give instructions, introduce the experimental manipulation, and or collect the data. Cobanoglu, Warde, and Moreo (2001) estimate that marginal, unit costs for postal mail survey are $1.93, compared to a marginal cost of close to zero for a Webbased survey, although fixed costs for the Web are higher. The differentials are much higher for interviewer-administered surveys (telephone or face to face), as one is paying for interviewers’ time for every contact attempt and completed interview. Practitioners estimate that the per-completed interview costs for telephone surveys range from $40 to well over $100. Consider how Web surveys are changing the nature and economics of questionnairebased research. With conventional, paper-based questionnaires, transcription of survey answers is an expensive and potentially error-prone process. The questionnaires themselves are relatively inflexible, either forcing a common sequence of questions for all respondents or requiring confusing instructions for skipping blocks of questions (Dillman, 2000). Survey organizations have long used computer-assisted interviewing APA-Internet Version 3.3 9/30/2003 Page 6 (CAI) for both in-person or telephone interviewing to overcome these problems (Couper & Nicholls, 1998). Interviewers enter data as they ask questions, and the software can customize the next question based on prior answers and other considerations. Internet surveys provide similar advantages to CAI systems, while eliminating the interviewer. Many software packages now exist that can create complex online questionnaires, where the data are written directly to a database. (See Crawford, 2002 for a review. http://www.asc.org.uk/ maintains a list of software for online surveys). Using these techniques, the researcher has greater control over the data-collection setting compared with executing a mailed survey. The researcher, for example, can constrain response alternatives with menus or dialog boxes and conduct checks as the questionnaire is being completed to identify missing or inconsistent data. By requiring respondents to submit their surveys incrementally, the researcher can obtain partial data even from those who fail to complete an entire questionnaire. This helps the researcher obtain a measure of biases in the sample and systematic differences between those who complete the survey and those who drop out. Automation also means that the assignment of subjects to experimental conditions within a questionnaire is a trivial exercise. The assignment can be based on subject characteristics or on responses to earlier items. The possibility of control and the potential size of the subject sample allow researchers to conduct large and complex experiments within a single study (See Sidebar 2). User metrics such as response latencies, changed answers, backing up, or other behaviors can be captured, permitting richer analysis of the process of the experiment and variations in its execution across subjects. The Implicit Attitude Test, described in Sidebar 2, uses reaction times to measure attitudes more subtly than traditional verbal attitude measures. Examining new social phenomena Up to this point, we have emphasized some of the opportunities of using the Internet as a research modality to increase the efficiency of studying traditional psychological phenomena. The Internet is also an important phenomenon in its own right. Like the telephone, television, and automobile before it, personal computers and the Internet are new technologies being adopted by a majority of Americans, with the potential to change the way they live their lives. Just as psychologists have long been interested in the way that television influences child development, prejudice, and violent behavior (Huston et al., 1992), so too psychologists are now examining the impact of the Internet (e.g., R. Kraut et al., 1998; McKenna & Bargh, 2002; Wellman & Haythornthwaite, 2003). The Internet is used extensively for interpersonal communication. Starting with landmark research by Hiltz and Turoff (1978) and by Sproull & Kiesler (1991), psychologists have examined how computer-mediated communication differs from other communication modes in influencing social interaction. More recently, psychologists have been especially interested in the longer-term impact of computer-mediated communication. They examine how time spent on email and in chat rooms contrasts with other Internet applications and its impact on social involvement and its psychological consequences (e.g., Kraut et al., 2002; McKenna, 1998). APA-Internet Version 3.3 9/30/2003 Page 7 The Internet is also the location for psychological and social phenomena that, if not entirely new, are rare in other settings. For example, although distributed work has existed for centuries (O'Leary, Orlikowski, & Yates, 2002), highly interdependent workgroups whose members are geographically distributed are a relatively recent phenomenon, made possible by improvement in computing and telecommunications, including the Internet. These new forms of working have caused researchers to reexamine how shared context and trust, often taken for granted in face-to-face settings, have their influence on group performance (e.g., Olson & Olson, 2000; Rocco., 1998). The challenge of designing ways to improve coordination and communication forces us to rethink conceptions of the world. For example, researchers are now deconstructing the concept of face-to-face interaction, to understand how its individual components can influence communication (e.g., Kraut, Fussell, Brennan, & Siegel, 2002). Others have examined the nature of commitment to very large groups (e.g., Moon & Sproull, 2000). Yet others have examined how the Internet allows individuals to assume and play with alternate personal identities, which may differ from their real-world persona in gender, age, or other normally static properties (e.g., Turkle, 1997). Challenges of Internet research: Data quality The preceding section highlighted the ways in which online research can reduce the cost of psychological research on traditional topics and open up new phenomena to the psychologist’s lens. These opportunities sometimes entail risks to both the quality of the research itself and to the human subjects who participate in it. In this section we discuss concerns about data quality associated with conducting research online.